Repository navigation
fix(captcha): cancel timed-out worker races - #81
TriDefender merged 6 commits into
Conversation
Propagate an AbortSignal from the CAPTCHA pool through the solver to worker dispatch, and abort/terminate parallel worker solves when the race deadline expires. Covers warm workers and worker-per-solve workers, with a hung-worker regression test.
|
Btw, after doing this change, I'm not seeing any more captcha failures. |
TriDefender
left a comment
There was a problem hiding this comment.
Verified on a local checkout of ea92cea: bun x tsc --noEmit clean, all 74 captcha tests green. I also ran ad-hoc probes against real worker fixtures to exercise the abort wiring end to end.
The core direction is right and mostly solid — the retry-ladder stop in solveFresh, not spreading signal into the worker postMessage, no in-process degrade on abort, no noteWarmDeath on abort-kill, and clearing the deadline timer on success are all handled correctly. Worker-per-solve mid-flight abort rejects in ~124ms (vs the 30s worker timeout) and warm mid-flight abort in ~4ms in my probes.
Two should-fix items before merge:
1. [P2] Stale abort listener in dispatchToWarm can kill an unrelated take's in-flight solve
The abort listener is registered {once: true} but is never removed when the dispatch ends through a normal (non-abort) settlement — the ok:false reply path, the 30s timeout path, or the message handler. solveInWorker removes the listener in its settle helper; dispatchToWarm has no equivalent.
Repro (real worker fixtures, verified end to end):
- Take A's warm dispatch fails with
ok:false(e.g. F008 velocity) → the handle returns to the pool idle, but dispatch #1's listener stays attached to A's signal. - Take B acquires the same idle handle and hangs on its solve.
- Take A's race deadline fires → A's stale listener runs
killWarm(h)unconditionally → B's in-flight worker is terminated and B rejects with A's deadline reason, whileb.signal.abortedis stillfalse(asserted in the probe).
Impact is bounded but real on Android (warm pool of 3; desktop default ZCODE_CAPTCHA_WARM_WORKERS=0 is immune): under exactly the hung-worker storm conditions of #82, handles churn fast and takes overlap, so this is reachable — and the killed solve surfaces in solveFresh as noteMintFailure("captcha take deadline"), polluting the mint-storm counters (8 such failures in 5min with no successes → false storm → telegram IP reset).
One-line guard (verified: eliminates the repro without affecting legitimate mid-flight aborts):
const onAbort = () => {
// Stale-listener guard: if this dispatch already ended (ok:false reply,
// timeout), the handle may have been re-acquired by ANOTHER take — a late
// abort from the original signal must not kill that solve.
if (h.pending?.id !== id) return;
killWarm(h, req.signal?.reason instanceof Error ? req.signal.reason.message : "captcha solve aborted");
dropFromPool(h);
reject(req.signal?.reason instanceof Error ? req.signal.reason : new Error("captcha solve aborted"));
};Alternatively mirror solveInWorker's removeEventListener in a settle-style completion path.
2. Regression test does not pin the mid-flight termination
"aborting a solve terminates a hung worker immediately" aborts synchronously right after calling solveViaWorkerOrInProcess, before the async entry resolution resumes — so it only exercises the throwIfAborted pre-check. Evidence: with both addEventListener("abort", ...) registrations deleted, this test still passes in 0.25ms (no worker is ever spawned), while a probe that spawns the worker, waits ~120ms, then aborts hangs without the wiring.
That means a future refactor could silently drop the onAbort wiring and this test stays green — issue #82 would regress without a signal. Suggested shape:
const pending = solveViaWorkerOrInProcess({ ..., signal: controller.signal });
await new Promise((r) => setTimeout(r, 100)); // worker spawned, job dispatched
const t0 = Date.now();
controller.abort(new Error("request deadline"));
await expect(pending).rejects.toThrow("request deadline");
expect(Date.now() - t0).toBeLessThan(5_000); // must not ride the 30s worker timeoutplus a warm-pool variant (ZCODE_CAPTCHA_WARM_WORKERS=1) asserting the killed handle is dropped from the pool and the next solve respawns. Also note the test lives in the "warm worker pool" describe but deletes ZCODE_CAPTCHA_WARM_WORKERS and exercises worker-per-solve — worth relocating or commenting while touching it.
Minor (non-blocking)
- The in-process fallback (
solveTraceless) ignores the signal. That's acceptable — a main-thread solve cannot be terminated and is bounded by its own timeouts — but worth a one-line comment stating cancellation covers the worker faces only, so nobody assumes full coverage. - Pre-existing, out of scope: a racer that completes after the deadline has its first token dropped rather than banked (
solveRaced'sserved/resolvepath).
Fix #1 (one line) and reshape the test per #2, and this is approve-worthy — the cancellation mechanism itself works exactly as described in the PR body.
…tests Skip the abort handler in dispatchToWarm when the dispatch already ended, so a late abort cannot kill an unrelated take that re-acquired the handle. Reshape the regression test to abort after the worker is dispatched, add a warm-pool variant, and note that in-process solves are not cancellable.
|
Thanks for the thorough review, and for the repro. Both items are addressed in the three new commits. 1. Stale abort listener in 2. Regression test (eaeb5c3, 1ecb6b3)
Validation: I ran the repo's |
TriDefender
left a comment
There was a problem hiding this comment.
Re-verified at 1ecb6b3 on a fresh checkout: tsc --noEmit clean, full bun test 1223 pass / 0 fail, matching your fork CI.
Both items are properly resolved:
1. Stale listener guard — if (h.pending?.id !== id) return; in onAbort is exactly right. I re-ran my original repro end to end against your head (take A settles ok:false → handle re-acquired by take B → A's signal aborts late): take B stays pending and unaffected, then rejects cleanly when the pool is stopped. The guard also passes through legitimate mid-flight aborts unchanged (~4ms rejection on the warm path in my probes).
2. Regression tests now pin the wiring — I deleted both addEventListener("abort", ...) registrations locally and re-ran the abort-test scenario: without the wiring it cannot reject quickly (rides the 30s worker timeout; the test hits bun's 5s timeout), so a future refactor that drops the wiring now fails CI instead of passing in 0.25ms. Good catch on the fixture, too — the original hanging-worker.mjs had no message listener, so the worker's event loop emptied and it exited immediately; with parentPort.on('message', () => {}) it genuinely hangs and the test exercises a real mid-flight termination. The warm variant's spawn counting (delta = 2 across two aborted solves) correctly pins both the mid-flight kill and that the dropped handle forces a respawn.
Two non-blocking notes, no action needed:
- There's no direct regression test for the stale-listener scenario itself (
ok:falsesettle → re-acquire → late abort from the first signal). The guard is one line and I've verified it live, so I'm not asking for one — just noting it if you ever touch this code path again. - The in-process fallback comment from eaeb5c3 ended up removed by bff1dfa. Fine either way.
Nice work — this closes both should-fix items cleanly. Approving.
|
Follow-up on your non-blocking note:
Checked on my fork's CI: all 1224 tests pass at 4079a45. With the guard removed, only the new test fails, so it pins the fix. |
Problem
CAPTCHA solves were failing in a cascade: a worker hung and hit the 30000ms worker timeout, the pool retried (four attempts), and parallel solves that had already lost the client-facing race kept running. Orphaned workers piled up, so later solves also failed and requests surfaced upstream 503s.
Fix
AbortSignalthrough captcha pool → solver → worker dispatch.captcha-worker-dispatch.test.ts.Scope
Only
src/proxy/captcha-pool.ts,captcha-solver.ts,captcha-worker-dispatch.tsand its test. No config or API changes.Validation
bun testpasses and the Docker image builds from this change (run on my side before opening the PR).Fixes #82